The Anglo-Norman Hub Text Corpus Digitisation Project

Challenges and Solutions

The nature of the challenges facing a digitisation project depends on the goals of that project. We can loosely define a generic goal as follows: 

Generic Goal of Any Digitisation Project

•	To provide a set of related resources to a designated user community in the format which best meets that community’s requirements

The challenges faced will depend on the specific goal of the project. The following discussion is specifically about the Anglo-Norman Hub Project with some reference to other approaches to text digitisation.

There are two fundamentally different approaches to text digitisation. Which is adopted again depends on the project's goals:

Choice of Output Medium for Digitisation of Books

•	Facsimile text image (adopted by many library digitisation projects)

•	Marked-up text (adopted by Anglo-Norman Hub)

Text images can be produced relatively quickly and easily provided that the scale of the project justifies the outlay on imaging equipment and digital storage, but their is a trade-off here between visual authenticity on the one hand and cross-referencing capability and legibility on the other:

Marked-up text resolves the cross-referencing issue, but entails a considerable overhead and may not always guarantee readability:

Specific Goal of the Anglo-Norman Hub Text Digitisation Project

•	To provide a set of digitised texts and articles to mediaeval linguists and historians which is searchable and fully cross-referenced within itself and to and from the Anglo-Norman Online Dictionary

The specific goal of the Anglo-Norman Hub project has a prerequisite of cross-referencing: consequently marked-up text is the only possible approach.

Given this decision, we can now formulate the three main challenges facing the project, and which are specific to the project:  

Main Challenges facing the Anglo-Norman Hub Project

•	Image to text migration for maximum throughput at minimum cost

•	Application of markup suitable for rendering and full cross-referencing

•	Handling of non-standard character sets (mediaeval abbreviations)

We will now examine each of these three challenges and how to manage them.

Image to Text migration.

There are two possible methods. Both require proofreading and both allow the preservation of appearance information.

Image to Text Migration Strategies

•	Optical Character Recognition
•	Re-keying

•	Both require subsequent proofreading
•	Both allow insertion of appearance metadata as provisional markup

The relative speed of OCR is offset by the high overhead of verification, especially for medieval languages.

Also, the possibility of providing detailed instructions on how to represent non-standard characters is critical in some cases.

Economic Image to Text Migration: Conclusions

•	Re-keying is more economic for the bulk of the mediaeval-language material
•	OCR is competitive for modern languages (critical material)
•	OCR can also be used for mediaeval language material 
1)	where good image quality is easily achieved
2)	with only standard characters
3)	when required by workflows

Re-keying is thus more appropriate for most of the material, but OCR can be used for the English.

There is also scope to vary the strategy to ensure continuity of throughput, providing these three conditions (at the bottom) are met:

Markup requirements

Markup requirements: must

•	Conform to widely-accepted standards
•	Be capable of encapsulating diverse document structures
•	Allow for automation
•	Enable internal and external referencing
•	Preserve as much appearance metadata as possible 
•	Not be tied to any one approach to rendering

Adoption of standards is essential to ensure continuing functionality across the internet. The standard adopted must be flexible enough to cope with considerable format diversity, and allow for automation if a large amount of material is to be processed. Cross-referencing is vital while at the same time efforts need to be made so that the rendering can reconstruct as many features as possible of the original appearance if required.

XML is the only technology which meets these criteria and the Text Encoding Initiative has provided a scheme to standardise XML markup of texts. The problem of non-standard characters however remains.

Standards for Markup

•	TEI-conformant XML meets all criteria

•	Unicode character standard does not include all mediaeval characters

Document Structures

•	Texts
–	Verse
–	Prose 
–	Lists & Tables
•	Critical material
–	Introductions (conform to prose structures)
–	Notes (do not conform to above structures)

The verse texts need the line numbers encoded for cross-referencing to notes. These can be generated automatically. In the case of prose texts, some have pointers to notes which can be bloc-replaced by the XML equivalents; others have notes referenced by line numbers, but as the line-endings in the prose are not necessarily preserved when rendered in a web environment, the pointers have to be inserted manually.

Markup Density and Automation

•	Verse: medium density; can be automated
•	Prose: variable density; can be automated if footnote pointers present
•	Lists & tables: medium density; can be automated
•	Critical material: high-density; many cross-references; limited scope for automation

Cross-referencing of Critical Matter

•	Need to navigate from pointer to note

•	Need to navigate cross-references from critical material to specific points in the text or elsewhere in critical material

•	Achieved by use of target-id pairs

To illustrate this here is a verse sample with line ids:

Extract from XML version of 
“La Passiun de St. Edmund”

<lg n="316"><l id="L1261">A Deu del cel ad graciéd</l>
<l id="L1262">E al martir suvent a voéd</l>
<l id="L1263">Que si bel l'at delivréd</l>
<pb ed="folio" n="123a"/><l id="L1264" n="1264">De ço qu'esteit ainz encumbrét.</l></lg>

And here is the critical matter which refers to these ids using targets, and also references to examples elsewhere in the text (as here at the bottom):

Extract from XML version of 
“La Passiun de St. Edmund”

•	<note id="N1261-4" target="L1261" targetEnd="L1264">These lines present several problems: (a) <q lang="AN" rend="b">A Deu. . .ad graciéd</q> <ref target="L1261">1261</ref>. The verb <term lang="AN" rend="i">gracier</term>, occurring here with an indirect object, normally takes a direct object and does so in its other occurrences in the text: <ref target="L826 L943 L1132">ll. 826, 943, 1132</ref>. 

Additional Markup for Critical Material

–	<term>: Terms discussed may need to be linked to the Anglo-Norman Dictionary
–	<q>: Citations: may need to be linked to their sources within the text base
–	<bibl>, <title> etc.: Bibliographical information needs to be encoded to link citations with their sources
•	Much of the above can be extrapolated from the appearance metadata embedded in the provisional markup
–	<hi>: to encode embedded appearance metadata whose significance is not apparent

The main point here is that the embedded appearance metadata is useful for identifying where such markup needs to be applied.

Here is one possible way of rendering this material, where you can see the citations in italics and links to take you to the text locations referred to:

“La Passiun de St. Edmund”
Rendered for a Web Browser

These lines present several problems: (a) A Deu. . .ad graciéd 1261 . The verb gracier , occurring here with an indirect object, normally takes a direct object and does so in its other occurrences in the text: ll. 826, 943, 1132 . T.-L. 4,502 cites one instance of gracier with indirect object, but in the construction gracier. qc. a qn . If this construction were applied here, ll. 1263-4 would have to be taken as the direct object of gracier and also, presumably, of voer 1262 . The use here of gracier with indirect object may have been influenced by the construction rendre graces a qn. employed at ll. 995, 1046, 1512 .

In conclusion the large amount of markup required necessitates automation wherever possible, which is facilitated by provisional markup applied at the previous phase.

Markup Requirements: Application

•	1,000 to 100,000 XML tags per document

•	Automation essential for high throughput

•	Digitisers can embed appearance metadata in provisional markup 

•	Well-designed provisional markup schemes facilitate automation

Representation of unusual characters.

Here a distinction must be drawn between transcription and transliteration. The first takes place at the digitisation stage, and the second takes place at the markup stage.

Handling of Non-Unicode Characters: 1) Transcription

•	Transcription is the one-to-one encapsulation of character appearance metadata

•	Transliteration is the expansion of abbreviated characters into an intelligible sequence of letters

•	Transliteration requires transcription as a starting point

•	Transcription codes must resemble originals to facilitate re-keying
•	

Here are some examples of medieval abbreviations and their meanings:

And here are some ways of encoding these. 

Here is an example illustrating the process of transcription:

At the transliteration phase automation is crucial but not straightforward. It has to be done on a word by word basis, which requires a concordance.

Handling of Non-Unicode Characters: 2) Transliteration

•	Manual transliteration would take too long

•	Blanket replacement is not possible because of ambiguous abbreviations

•	Semi-automated transliteration can be achieved using a list of words for block-replacement, derived from a concordance

•	The appearance metadata from the transcription should remain embedded

The concordance shows every context for every abbreviated word-form. 

In this instance all examples bar one are cases of the Latin Comitatus, the exception being (here) the French word Come:

Here is the transcribed text and the XML result which encapsulates the transcription data and shows the transliterations as text.  .  .  

And here is the XML with the final result, which is now readable Latin, rendered to show the presence of abbreviations by italics: